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ABSTRACT 


This thesis compares helicopter engagement ranges from 
an operational field test to similar ranges generated by 
simulation of the test in Janus(A). The purpose is to analyze 
the feasibility of accrediting the Janus(A) combat model for 
the Post-Test Modeling Phase of an Army concept called Model- 
Test-Model. Means and distributions of helicopter engagement 
ranges are analyzed. The Janus engagement ranges are greater 
than those of the operational test. No common link between 
the two tests and the scenarios is apparent. Other issues 
include time taken to ensure that a credible database is 
entered in Janus and improvement of terrain and helicopter 


representation for a more realistic result. 
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I. INTRODUCTION 


A. GENERAL 

The goal of operational testing is to "conduct field 
testing, under realistic combat conditions, of any item of 
weapons, equipment, or munitions for the purpose of 
determining the effectiveness and suitability for use in 
combat by typical military users" [Ref. 1]. Unfortunately, 
due to budget constraints and the lack of maneuver area, 
equipment, troops and time, it may not be feasible to test 
these items thoroughly. The use of modeling and simulation 
can help to close the gap when full scale testing is 
unavailable [Ref. 2:p. 1]. Janus(A) is one of the models used 
for this purpose. 

Comparison between operational field tests and high 
resolution combat models such as Janus can provide more 
information to assist test design and possibly extend test 
procedures. Once a field test has been performed, the data 
obtained can be used to calibrate the model to the test. 
Calibration entails making adjustments to input data or model 
logic to obtain a closer agreement with an external index 
[Ref. 3:p.5]. If the model proves to be credible, it can be 
used to extend test results beyond the field test environment, 


thus saving time and money which would be required to run 


further field tests. The Army uses a concept called Model- 
Test-Model (M-T-M) to implement modeling and simulation in 


operational testing. 


B. THESIS OBJECTIVE 

This thesis analyzes the feasibility of accrediting the 
Janus combat model for the post-test modeling of aircraft 
engagement ranges. Specifically, helicopter engagement ranges 
collected from the Line-of-Sight Forward (Heavy) (LOS-F-H) 
Initial Operational Field Test and Evaluation (IOTE) conducted 
at Fort Hunter Liggett, California in Spring 1990 are compared 
to similar ranges generated by simulation of the test in the 
Janus combat model. Route data collected from the operational 
test are replicated within Janus so that runs can be made to 
obtain model engagement ranges. The means and distributions 
of engagement ranges from both the model and operational test 
are compared statistically to analyze the feasibility of using 
the simulation for future post-test modeling of the LOS-F-H 


system. 


C. MODEL ACCREDITATION 

Model accreditation is part of the process of validating 
or establishing credibility of the model. Validation is the 
"process of determining that a model is an accurate 
representation of the intended real-world entity from the 


perspective of the intended use of the model" [Ref. 3:p. l: 


Enclosure 2]. Accreditation is necessary if the model is not 
fully validated. It is the process of certifying that a model 
achieved an established standard when applied for a specific 
purpose [Ref. 2:p. 6]. Models are accredited for particular 
types of applications since validation is a continuous process 
and full validation may not be technically or economically 
feasible. A model is subject to accreditation when it is 
proposed for use with a new application or system [Ref. 3:p. 
WWE In this case, the Janus combat model requires 
accreditation with the new LOS-F-H system performing 


helicopter engagements. 


D. DEFINITION OF ENGAGEMENT 

An engagement is defined as the moment the fire button is 
pushed on the LOS-F-H system. Of primary interest is the 
range between the target and the system when engagement 
occurs. This analysis will be concerned only with the range 
of first engagement. This is the initial range at which the 
gunner shoots at a target for the first time. Consecutive 
shots by the same system on the same target will not be 
considered. These consecutive shots are assumed to be 
statistically dependent for the purpose of this thesis. Both 
Janus and the LOS-F-H data will be edited to provide the range 


of first engagement. 


E. LINE-OF-SIGHT FORWARD (HEAVY) SYSTEM [Ref 4:p. 1-7] 
Recently, the Army has proposed a major revision in 
divisional air defense concept. The Forward Area Air Defense 
System (FAADS) program integrates five components designed to 
protect a division from low altitude air threat. One of the 
primary components in a heavy division is the LOS-F-H air 
defense system. The LOS-F-H is a surface to air missile 
system mounted on a modified Bradley fighting vehicle, manned 
by a crew of three: a commander/radar operator (RO), a 


gunner/electro-optics (EO) operator, and a driver (Figure l). 


Y bh 


— 


OU 
y 


VEHICLE DRIVER 


RADAR OPERATOR 


EO OPERATOR (SQUAD LEADER) 
M3A1 BRADLEY CHASSIS (GUNNER) 





Figure 1 LOS-F-H System 


The fire unit has its own acquisition and tracking radar 
which is capable of tracking several aircraft while scanning 
for others out to 20 kilometers. The system carries eight 
missiles which weigh 112 pounds each and are in ready-to-fire 
Canisters. Each missile is a laser beam rider guided by a 
coded laser beam which’ minimizes the effects of 
countermeasure. The unclassified range of the missile is 8 
kilometers. 

The crew uses a Forward Looking Infrared sensor (FLIR) or 
an optical (TV) sensor to help identify and track the target. 
An Identification, Friend or Foe (IFF) device which receives 
an encrypted transmission sent by friendly aircraft is used to 
make final identification of an enemy aircraft. The gunner 


then launches a missile and tracks it to the target. 


F. MODEL-TEST-MODEL CONCEPT 

There are three phases to the Model-Test-Model (M-T-M) 
concept: pretest modeling phase, field test and modeling 
phase, post-test modeling phase [Ref. 5:p.3]. 

1. Pretest Phase 

This phase is designed to assist planners prior to 

actual operational testing. Selected models exercise possible 
test scenarios to provide information on the best possible 
test design. This information includes methods to optimize 
data collection, minimize test failure and alert testers to 


the impact of external constraints. The results provide 


planners confidence that test objectives can be achieved and 
provide the modeler important information that can be used 
with future modeling of the system. This phase can save the 
designer valuable time which would otherwise be wasted in the 
field [Ref 6]. Unfortunately, this phase was not conducted 
prior to the field test for the LOS-F-H. 
2. Field Test and Modeling Phase 

This phase involves a series of complete field tests 
to evaluate the new system and model runs to provide rapid 
analysis and feedback to test personnel. Operational tests 
are run in order to provide an assessment of how the system 
characteristics perform in a variety of different operational 
scenarios. Under the best circumstances, scenarios are 
developed in the pre-test phase for use in the initial 
operational trials. Field testing is done by military 
personnel ina series of trials conducted to replicate the use 
of the system. These trials are usually between two opposing 
forces. Once a field test is complete the data is passed to 
the modeler who performs successive iterations of model runs 
to provide feedback, but most importantly to calibrate the 
model code and model data [Ref 5]. This involves updating 
input parameters such as weapon characteristics to field 
constraints. Actual play positions are used as input to 


complete the model runs and a comparison analysis is conducted 


at the individual event level (i.e. engagements) to determine 
the possibility of accreditation. 
3. Post-Test Modeling 

The third and final phase involves the cautious use of 
the ‘calibrated’ model in order to extend the test results to 
conditions, situations and threats not tested in the field 
fRef 5]. This ‘non-testing’ may be due to cost, safety, 
environmental, equipment or some other type of constraint. 
During this phase it is important not to extend the model 
beyond the point at which the - calibrated eeesae 


representations would no longer be valid. 


II. OPERATIONAL FIELD TEST 


A. BACKGROUND 

The operational field test of the LOS-F-H system consisted 
of fifty maneuver trials. These trials were conducted from 9 
April to 23 May 1990. Each trial was a force on force battle 
which generally lasted one hour. Both day and night trials 
were performed, as well as trials in MOPPO and MOPP4 (two 
variations of Mission Oriented Protective Posture). The 
battles involved Red and Blue mechanized forces of battalion 
(Minus) strength. Battalion (minus) indicates that only part 
of the battalion was used. Surrogate aircraft representing 
the Mi-24 Attack Helicopter (HIND), the Mi-14 Medium 
Helicopter (HAVOC) and the Mi-8 Medium Helicopter (HIP) 
supported the Red forces. The LOS-F-H mission was to defend 
the Blue maneuver force against air attack as the force 
conducted its mission. Different scenarios were tested with 
variations in offensive and defensive operations and Blue 


force orientation. 


B. AIRCRAFT PRESENTATION 

Tactical and safety controls were provided for all 
aircraft. Airspace safety control was of primary concern and 
sometimes dictated flight routes, altitudes and tactics. The 


maximum altitude above the terrain for all helicopters was 150 


feet, but there was no minimum altitude. Helicopters entered 
the battle area making their approach to the Blue player’s 
Pront. It was assumed that Blue players received lateral 
protection from notional units which are units that are not 
physically on the ground, but are perceived to be there by all 
players. Aircraft were accounted for in presentations. A 
presentation began when an aircraft left its holding area to 
enter battle and ended when it returned to its holding area. 
If an aircraft was ’killed’ during a presentation, it would 
return to its holding area and would be revived by the 
controllers and sent into battle as another aircraft. 
Aircraft were the only players revived during the trial and 
that occurred only in holding areas [Ref. 4:p.2-22]. 
Presentations lasted anywhere from 7 to 30 minutes depending 


on the trial and number of aircraft desired to represent. 


C. DATA COLLECTION 

Firing information for all weapon types were provided by 
a laser installed on each firer and by laser sensors on each 
target. A laser pairing provided the real-time computer with 
the firer and target identification. This process was called 
Real-time Casualty Assessment (RTCA). The RTCA consisted of 
firer and target identification, weapon type, trigger pull 
time, probability of kill, assessment of shot and time of 
impact. Video tapes were used inside the LOS-F-H fire unit to 


record the battle and a Range Measuring System (RMS) recorded 


the system location data as well as engagement range 
information using a form of triangulation [Ref 4]. 

Several computer reports were generated providing 
information for analysis. Two of these files, the Player 
Position Location File (PLS) and the Attack Engagement File 
(AEF) provided information specifically concerning the 
LOS-F-H. The PLS file recorded player locations at every 
second during the battle. This data was used to imitate the 
actual battle in Janus. The AEF file provided information on 
the trial engagement segment, specifically the LOS-F-H 
engagement of aircraft. The data from this file were used for 
engagement range comparison with the data obtained from the 


Janus runs. 


D. DATA LIMITATIONS 
1. Range Measuring System (RMS) Errors 

This type of error affected the vehicle and aircraft 
routes as well as engagement ranges. The RMS at Fort Hunter 
Liggett records the position data and calculates’ the 
engagement range from the position data. Errors due to 
inaccurate triangulation (spikes) or lost signals (gaps) could 
provide false location and engagement ranges. Although most 
of the data was smoothed there was still some error associated 
with this problem. This error can affect the actual player 


location as well as the calculated engagement range. 
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2. Real-Time Casualty Assessment (RTCA) Errors 

RTCA missed about 18 percent of the LOS-F-H launches 
because of low or dead batteries in the laser equipment. 
Other launches resulted in no laser pairing and therefore 
provided no target. These launches were reconstructed as 
closely as possible by analyzing video tape and using player 
location plots. Over 90 percent of the total launches were 
evaluated by analyzing video and verifying RTCA results 


leaving almost ten percent of the RTCA results not verified. 


E. DATA SELECTION 

Although the test consisted of fifty trials, only five 
were selected for analysis. The five trials chosen were under 
identical conditions of daytime, MOPP0, no smoke, Blue in 
defense and facing northwest. These trials were selected 
Since they provided the most helicopter engagement data 
(Table 1). 

TABLE 1 TRIAL CONDITIONS 


LOS-F-H TRIALS SELECTED: 
100B, 112B, 122B, 123B, 125B 


FACTOR CONDITION 
| TIME DAY 


PSMORE NOE 
[cuemrcaL NONE 





il 


III. JANUS 


A. BACKGROUND 

Janus is an interactive, brigade level, two sided, event 
driven simulation that models fighting systems as entities 
(helicopter, tanks, etc.). Entity characteristics include 
descriptions of weapons carried, weapon capabilities, movement 
speeds and how they are affected by terrain, ammunition and 
fuel, crew performance, sensor data and supply/resupply data. 
The data can be interactively reviewed and changed by the 
user. As with any simulation an accurate and complete 
database is crucial to operation and output. There are a 
large number of interconnections between portions of the 
database; therefore, altered data in one area may affect the 
outcome of the simulation. Terrain is depicted with contour 
lines, vegetation and cities. Each terrain cell has a fifty 
meter resolution which corresponds to the Defense Mapping 
Agency elevation, vegetation and cultural feature description. 
Graphical symbols represent one or more systems and each 
system can have one or more weapon [Ref 7]. 

A Night Vision Electro-Optical Laboratory (NVEOL) model is 
used for detection. Engagement results are then determined 
using comparison of random number draws to a probability of 


hit and kill database. An extensive postprocessing procedure 
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allows for collection of data such as detection and 


engagements [Ref 8]. 


B. TEST DATA CONVERSION 

The PLS files from the five field trials were converted 
into appropriate Janus databases to replicate the field trials 
vehicle and helicopter movement. A FORTRAN program designed 
by Captain Al East, a former student at the Naval Postgraduate 
School, was revised to read the PLS data files and arrange the 
data into a National Training Center (NTC) format [Ref 9]. 
Another FORTRAN program (INITNTC) written by Mr. Al Kellner, 
a programmer from TRADOC Analysis Command (TRAC) White Sands 
Missile Range, was used to convert the NTC format into Janus 
format. This conversion process creates scenarios that 
replicate the force structure and vehicle routes of the field 
trials. At this point the modeler has the option to adjust 
some input parameters in order to allow the simulation to 


represent the field trial with greater accuracy. 


C. INPUT PARAMETERS 

The Janus database was not changed for weapon and system 
characteristics. The data was previously entered by students 
at the Naval Postgraduate School in conjunction with thesis 
work and is assumed to be accurate. Since the altitude of 
the helicopters in the field trials was not accurately stated 


in the Test Report [Ref 4], and the analyst was not present, 
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there is uncertainty in the flight altitudes. To test the 
sensitivity of altitude change, the helicopters altitude was 
changed and each trial was run with three different altitude 
levels: 15, 25, and 35 meters. Therefore, the five Janus 
trials became fifteen Janus trials. These altitudes were 


chosen based on the analysts experience as a pilot. 


D. DATA COLLECTION 

The fifteen Janus trials were each run three times. This 
was done to provide enough engagement range data for analysis. 
Each scenario was run interactively at first to insure that 
the simulation and the entities were behaving properly. The 
three runs provided a total of 10 to 50 helicopter engagement 
ranges for each trial. The Janus postprocessor was used after 
each trial to provide a cumulative list of engagements. This 
list was manually reviewed to provide only helicopter first 


round engagements. 


E. DATA LIMITATIONS 

The PLS files did not account for the repeated 
presentations of helicopters in the field trial. This created 
a problem with the converted data. If a helicopter made more 
than one presentation, the PLS file only showed one long 
continuous route. This made it necessary to manually generate 
the aircraft presentations and routes in Janus. Since ninety 


percent of the helicopter routes were entered manually, there 
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may be errors in their location. The Janus screen provides 
ten digit grid coordinates, while the human can not really 
distinguish to this accuracy. This can cause some error when 
engagement ranges are determined by Janus. If the location of 
the aircraft is incorrect then the engagement range will be in 


error. 
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IV. ANALYSIS 


A. GENERAL 

The purpose of this comparison is to determine if the 
means and distributions of engagements are the same for the 
field test and Janus. If they are different then a possible 
trend which could indicate a shift in location or a possible 
error might be of interest. Samples from the field test 
consisted of first range engagements from the five trials. 
Samples from Janus consisted of three runs of each trial for 
each helicopter altitude 15, 25, and 35 meters. There was a 
total of 45 runs. The statistical software Statgraphics [Ref 


10] was used to conduct the analysis. 


B. ALTITUDE SENSITIVITY 5 


Since the actual altitudes flown by the helicopters were 
not known, it was of interest to see if the altitude 
adjustment used in Janus would provide sufficient differences 
to affect the analysis. This sensitivity analysis could help 
to determine the best set of data for comparison with the 
field trials. Each scenario was run with the three different 
helicopter altitudes and the results were compared. Three 
assumptions were made concerning the Janus data for this 
analysis. The data was assumed to be normally distributed, 


each sample is a random sample from its respective population, 
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and data in each sample are independent of data from the other 
samples. The medians were then compared using a notched box 
and whisker plot (Figure 2) and then a One Way Analysis of 
Variance (ANOVA) was done to test the null hypothesis that the 


means were equal (Table 2). 
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Figure 2 Notched Box and Whisker Plot 
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TABLE 2 ANOVA VALUES FOR JANUS TRIALS 


ee 





*.0700 


*.. 9566 


*.6461 


The notched box plot provided an approximate 5% test of the 
null hypothesis that the true medians are equal. Since the 
notches overlap, we fail to reject the null hypothesis [Ref 
11]. With the ANOVA analysis we failed to reject the null 
hypothesis with a 0.05 significance level. This provides no 
Support for sensitivity to a change in altitude (for these 
three altitudes) [Ref 12]. For this reason, the central 
altitude of 25 meters was chosen for use in the continuation. 
It had the most data points. A summary of statistics for the 


Janus (25 meter) data and the field data are in Appendix A. 


C. ANALYSIS OF NORMALITY [Ref 12] 
Both the field data and Janus data were compared to the 
normal distribution (Figures 3 and 4) via Quantile-Quantile 


plot . 
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Figure 3 Field Data VS Normal Probability Plot 
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The field data appeared close to normal, but the Janus data 
showed greater deviation. This was especially true at the 
higher range end of the Janus data. This could be due to the 
fact that Janus would not engage helicopters beyond the range 
capability entered in the database. It seems unwise to treat 
the data as normal because of these deviations from straight 
lines. A less formal view of the data appears in the Notched 


Box and Whisker Plot (Figure 5). 
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—— rl ee 


This shows at a glance that only two of the trials (100 and 
122) appear to have equal medians using a 95% confidence 


interval. 


D. NONPARAMETRIC ANALYSIS [Ref 13] 

Nonparametric analysis was conducted since the field and 
Janus data could not be supported by the normal distribution. 
The assumption that the data was Independently Identically 
Distributed was made pertaining to the following tests. Both 
the Kruskal-Wallis and Mann Whitney test were used to test the 


null hypothesis that the means are equal (Table 3). 


TABLE 3 KW/MW P-VALUES 





*.29133 | 
00136 | 


* 27954 
i 00001 


~JU0NZ7 


The table again indicates that trial 100 and 122 fail to 
reject the null hypothesis with a 0.05 significance level. 
These results agree with the results found using the t-test 


and ANOVA. 
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The two distributions were then compared using the 
Kolmogorov-Smirnov Two-Sample Test (Table 4). This test was 
used to test the null hypothesis that the distributions were 


the same. 


TABLE 4 KS TWO-SAMPLE TEST VALUES 





KS TWO-SAMPLE TEST 


We fail to reject the null hypothesis with a 0.05 significance 


level for only two trials 100 and 122. This provides 


additional support for the previous results. 


E. FURTHER ANALYSIS 

The above analysis gives us the notion that some, but not 
all, Janus runs are comparable to the field trials. Let us 
search for a possible trend that may indicate the reason. 

1. Trimmed Sample 

Janus trial 125 has the most outliers (Figure 5), so 

this trial was chosen for application of a t-test with trimmed 
data. It is possible that some extreme points in the Janus 


data were influencing the mean and distribution to shift, and 
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therefore leading to the rejection of the null hypothesis. A 
10% trimmed sample was compared to the field data with the 
null hypothesis that the trimmed mean equals the field data 
Mean. The result was hand calculated and a p-value of .00234 
was determined. This shows that even if the Janus data was 
trimmed the result remains to reject the null hypothesis with 
a 0.05 significance level. 
2. Quantile-Quantile/Regression Analysis 

Another attempt to compare the distributions was done 
using a quantile-quantile plot for both the reecenent range 
and time. The plot was computed based on interpolated 
quantile values for the Janus data [Ref 10:p. 55]. This gives 
us the ability to look at the specific difference in each 


distribution and for possible trend behavior (Figure 6 and 7). 
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Figure 6 Quantile-Quantile Plot of Ranges 
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Figure 7 Quantile-Quantile Plot of Times 
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The solid line is the x=y line. If the distributions were the 
same, the points would fall on this line. If the 
distributions are different and some trend is determined, 
scaling laws can be developed which would allow Janus data to 
be converted to determine field trial outcomes. 

The data points were then fitted with a simple 
regression line with some extreme points disregarded [Ref 14]. 
The dotted line on the graph represents this regression line 
and the circled points are the ones selected to be disregarded 
when formulating the regression equation. The regression data 


(Table 5) was computed based on: 


Qy; - Janus Quantile Engagement Range Data 
Qy> ~ Janus Quantile Time Data 


Qy, - Field Quantile Engagement Range Data 
Qyo - Field Quantile Time Data 


Oy; = a; + b; Qy) 


Oya = a2 + bo Oxo 











Tana aa 
| 2.3684 | -17.774 | .96152_| 
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There does not appear to be any trend or similarities in any 
of the quantile plots. Each regression equation varies 
dramatically indicating a difference due to scenario change. 
Some seem to be merely a shift in location, while others have 
a completely different distribution. There does not appear to 
be one specific transformation that could convert the Janus 
data to the results seen in the field. In other words, the 
scaling laws appear to be scenario dependent. 
3. Predicted Values Comparison 

A final attempt to find some link between the Janus 
output and the field trial was done using the simple 
regression equations formulated from the Q-Q plot. The raw 
data from the Janus trials was used to find predicted values 
for the field trial. The purpose was to seek a correlation 
between time of engagement and its range. Such a correlation 
might help connect the scenarios and the scaling laws. The 
scatterplots are displayed in Figure 8. There appears to be 
no pattern in these graphs, again giving us no specific link 


from Janus to the field environment. 
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V. CONCLUSIONS/RECOMMENDATIONS 


A. CONCLUSIONS 

At this time, Janus should not be accredited for post-test 
modeling of helicopter engagements by the LOS-F-H. In three 
of the five scenarios analyzed, significant differences exist 
between Janus and the field trial helicopter engagement 
ranges. Janus mean engagement ranges were higher in all five 
cases (Appendix A). This indicates that Janus had the Line of 
Sight (LOS) and was engaging targets prior to the actual field 
trial. Several reasons are offered for interpretation. 


1. The terrain database used in Janus has a fifty meter 
resolution. Because of this, many lone trees, small 
rolling hills and other obstacles are not replicated. 
These obstacles may make some difference in the LOS for 
the field trials. 


2. Most of the helicopter routes were entered into Janus 
manually. A slight inaccuracy in the route data could 
result in a helicopter being exposed in the Janus scenario 
while it was not in the field scenario. 


3. The Janus database was assumed to be correct. If some 
parameters were inaccurate it can affect the performance 
of the system by either increasing or decreasing its 
ability. 


4. The actual helicopter altitude was unknown since the 
analyst was not present at the field trial and the after 
action report only indicated a maximum altitude. The 
analyst used her experience and judgement to determine the 
helicopter altitude used in Janus. The three altitudes 
tested showed no sensitivity, but if a lower altitude was 
analyzed, this may not be the case. 


on. 


5. The Janus simulation will always fire at its first 
opportunity. This may not be the case in the field 
environment. Personnel in the operational test may not 
have always fired at their first opportunity. This may be 
due to several reasons which can be analyzed further, but 
are not in the scope of this thesis. 

The data collection limitations in the field test were minor 

and the analyst does not believe that they affected the 


results. 


B. RECOMMENDATIONS 

In the future, if the Model-Test-Model concept is to be 
integrated, modelers must be present through the entire test 
sequence. This will ensure appropriate interaction between 
the field test and model. The modeler will have no unanswered 
questions about aircraft altitude or other field trial 
specifics and can ensure that the appropriate data is being 
supplied by the field test. This will also test the models 
data base to ensure the systems are being represented 
accurately in the model. 

More analysis should be done on the Janus’ terrain 
database. Fifty meter :»resolution does not provide enough 
accuracy to replicate the field test. Future studies and 
analysis using higher terrain resolution is suggested. 

Although this analysis could not accredit the Janus model 
for modeling of helicopter engagements by LOS-F-H, the analyst 
believes that with the appropriate database and involvement 


the results would improve. 
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